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Abstract: Background: Every physician in Ontario needs to know how to fill out a Form 1 in order to legally hold 
a person against their will for a psychiatric assessment. These forms are frequently inaccurately filled out, which 
could constitute wrongful confinement and, in extreme circumstances, could lead to fines as large as $25,000. 
Training people to fill out a Form 1 accurately is a large task, and e-learning (Internet-based training) provides a 
potentially efficient model for health human resources training on the Form 1. Objective: In this study, we looked 
at the efficacy of an e-learning module on the Form 1 by comparing baseline knowledge and skills with posttest 
performance. Methods: 7 medical students and 15 resident physicians were recruited for this study from within 
an academic health sciences setting in Flamilton, Ontario, Canada (McMaster University). The intervention took 
place over 1 hour in an educational computing lab and included a pretest (with tests of factual knowledge, clinical 
reasoning, and demonstration of skill filling out a Form 1), the e-learning module intervention, and a posttest. The 
primary outcome was the change between pre- and posttest performance. A scoring system for grading the 
accuracy of the Form 1 was developed and two blinded raters marked forms independently. Participants were 
randomly assigned to one of two sequences of assessments (A then B vs B then A), with a balanced design 
determining which test the participants received as either the pretest or posttest. Inter-rater reliability was 
determined using the Intraclass Correlation. Repeated measures analysis of variance was conducted. Results: 
The Intraclass Correlation (ICC) as the measure for inter-rater reliability was 0.98. For all outcome measures of 
knowledge, clinical reasoning, and skill at filling out the Form 1 there was a statistically significant improvement 
between pretest and posttest performance (knowledge, F(1 ,21 ) 54.5, p<0.001; clinical reasoning, F(1 ,21 ) 9.39, 
p=0.006; Form 1 skill, F(1 ,21 ) 15.7, p=0.001). Further analysis showed no significant differences or interactions 
with other variables such as between raters, the order of assessment, or trainee type. Conclusions: Under 
laboratory conditions, this e-learning module demonstrated substantial efficacy for training medical students and 
residents on the theory and practice of filling out the Form 1 of the Mental Health Act. E-learning may prove to be 
an efficient and cost-effective medium for training physicians on this important medico-legal aspect of care. 
Further research is required to look at the longer-term impact of training and broader implementation strategies 
across the province for medical trainees and practicing physicians. 
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1. Background 

Every physician in the province of Ontario, Canada needs to know how to complete a Form 1 of the 
provincial Mental Health Act in order to legally hold a person against their will for a psychiatric 
assessment. Despite the medico-legal importance, medical students, residents and physicians in 
practice receive very little or no formal training in how to perform this task. These forms are frequently 
inaccurately filled out, which could constitute wrongful confinement and, in extreme circumstances, 
could lead to fines as large as $25,000. In our academic teaching hospital, an audit of Form 1’s from 
the Emergency Department, Internal Medicine and Surgical wards revealed that over 50% of the 
Form 1’s were filled out incorrectly (Levinson and Garside, unpublished data). 

To date, there is very little formal curriculum for undergraduate and postgraduate medical trainees in 
the content area of mental health law; as well as very few continuing medical education opportunities. 
This is striking given that several areas of the Objectives of the Medical Council of Canada (MCC) 
outline the importance of these topics (Medical Council of Canada 2009), and many of the other core 
MCC objectives reference these concepts. The increase in the aging population and high prevalence 
of hospitalized patients with dementia or delirium argues for the importance of training in mental 
health law issues. 
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Few published studies have examined gaps in physician performance or curriculum in this area, but 
those studies that have been done point to significant learning needs. A survey of Canadian 
emergency medicine residents identified learning needs in consent and capacity assessment (Pauls 
2006). A survey of US emergency medicine residents identified training in informed consent for 
procedures as an important need (Gaeta 2007). Studies in the UK have also revealed significant 
deficits in physician knowledge in mental health legal domains such as consent and capacity 
(Jackson and Warner 2002; Schofield 2005; Evans 2007). Clearly many of these important issues are 
rarely addressed in training programs, a fact that is highlighted by the pretest performance of 
postgraduate participants in the study. 

Training all physicians how to complete a legally valid Form 1 is a very large task. First, there are a lot 
of physicians and medical trainees in Ontario that require training: 23,266 fully licensed doctors; 5,185 
postgraduate trainees/residents; and 3,245 medical students (OPHRDC 2008). Other elements that 
complicate the training: some physicians fill out Form 1’s relatively frequently (psychiatrists, 
physicians who work in emergency department settings), vs. some who are only required to fill out the 
Form infrequently. The Form itself is not that onerous - it is 3 pages long; and the associated Form 42 
that must be provided to the patient in some circumstances is only 2 pages. However, the Form is not 
that intuitively designed and little-to-no time is devoted in training programs to this skill. 

E-learning (Internet-based training) provides a potentially effective, standardized, and cost efficient 
model for health human resources training. A recent meta-analysis and systematic review by Cook et 
al showed large effect sizes for Internet-based instruction with health professions’ learners (Cook 
2008). However, most of the studies in that comprehensive review looked only at knowledge 
outcomes, with very few reporting a skills outcomes. Cook et al found 16 of 130 studies (12.3%) 
comparing an e-learning intervention vs no intervention that reported a skills outcome, with a large 
pooled effect size of 0.85. 

There have been no studies to date that have examined the use of an e-learning module to teach 
physicians how to accurately complete the Form 1 . We hypothesized that an e-learning module would 
be an effective method to train physicians in this skill. 

2. Objectives 

The objective of this study is to examine the efficacy of an e-learning module on the principles and 
practice of filling out the Form 1 of the Ontario Mental Health Act by comparing baseline knowledge 
and skills with posttest performance. 

3. Methods 

3.1 Design 

The study was a prospective pretest/posttest design, balanced with respect to assessment order (see 
below), and the experiment was conducted in a single sitting. The total study time was approximately 
1 hour. 

3.2 Setting 

All participants performed the experiment in either a 25-station computer laboratory within our 
academic health science centre’s computer services unit or Internet-enabled classroom settings 
within the university. In both settings there was a study supervisor (author SK) whose interaction with 
the subjects was scripted through a pre-written protocol. Workstations in the educational computing 
lab were personal computers running the Windows XP operating system; in the Division of e-Learning 
participants used either a Windows PC desktop or laptop. Several sessions were conducted with 
different numbers of participants from the eligible cohorts from April-July 2007. Study sessions were 
proctored to ensure no additional communication between study participants until completion of the 
posttest. 

3.3 Participants 

All final year undergraduate medical students (clinical clerks, year 3 in the McMaster program) and 
residents from psychiatry, emergency medicine, internal medicine and surgery were eligible to 
participate. Recruitment efforts using email and poster advertisements were targeted among clinical 
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clerks during their psychiatry, emergency medicine and family medicine rotations; recruitment among 
residents focused on psychiatry, emergency medicine, surgery, and internal medicine residents. We 
excluded any participants who had previous exposure to the training module. The Michael G 
DeGroote School of Medicine within the Faculty of Health Sciences at McMaster University in 
Hamilton, Ontario, Canada has approximately 140 clinical clerks out of a total undergraduate medical 
school class of 420 students; and a postgraduate medicine program of approximately 1500 residents 
across all programs. Based on the fact that most pretest/posttest no intervention control studies show 
very large effect sizes of approximately 1 .0, we calculated a sample size of 20 participants using the 
simple formula of 16/effect size squared, using an estimated effect size of 0.9. 

3.4 Intervention 

The e-learning module was designed to provide physicians and medical trainees with a review of the 
legislation and the methods that are used to bring a person to hospital against their will in Ontario. In 
addition, key aspects of completing the Form 1 of the Mental Health Act were reviewed in detail, 
including the examination, criteria for dangerousness, concept of mental disorder, time-sensitive 
issues, and invalid Form 1's. An example of the Form can be accessed through the province of 
Ontario’s Ministry of Health and Long-Term Care (MOHLTC) website at 

http://www.health.gov.on.ca/english/public/forms/form_menus/mental_fm.html. The educational 
module was designed and developed by two psychiatrists with knowledge and experience in using 
and teaching about the Form 1 (SG and AJL), and a lawyer with expertise in mental health law (MB). 
The Instructional Systems Development process was used as an instructional design framework, 
including previous formative evaluation methods such as additional subject matter expert review, one- 
to-one feedback and field trials with medical students and residents (Dick 2005). The module followed 
best practices with regard to multimedia learning using multimedia where appropriate, worked 
examples, and practice cases (Clark and Mayer 2007; Mayer 2001). 

The module was designed to take between 30 and 45 minutes and contained the following contents: 

■ Introduction 

■ Associated resources (linking to the PDF of the relevant forms) 

■ Objectives 

■ Overview: how can a person be taken to a hospital in Ontario? 

■ Key points/concepts related to the Form 1 

■ Guided tour of proper completion of the Form 1 (multimedia, narrated Flash animation allowing 
the learner to have both a ‘guided tour’ or a ‘click and reveal’ self-directed exploration of the Form 
1 with comments 

■ Worked example of a properly completed Form 1 based on a written case study 

■ Common errors on the Form 1 , multimedia asset highlighting errors on the Form 1 

■ Summary of common errors review 

■ 8 practice cases using questions (either single- or multiple-select questions, with a clinical stem 
and 5 response options) with immediate expert feedback provided for each question 

■ Acknowledgements. 

The module was created using extensible Markup Language (XML) and output for HTML using XSL 
transformation. The output format was compatible with the international SCORM e-learning standard 
(Advanced Distributed Learning 2009). The module was hosted on our Division of e-Learning server, 
and compatible with Internet Explorer and Firefox browsers on Windows, as well as Firefox and Safari 
on the MacOS. 

3.5 Measurement 

Two different assessments were created, Assessments A and B. Each assessment had 5 multiple 
choice questions to assess knowledge (e.g. factual questions regarding mental health law 
procedures); 5 multiple choice questions to assess clinical reasoning using clinical scenarios and 
asking participants to identify specific indicators/criteria that would suggest appropriateness of using 
the Form 1 ; and 1 clinical scenario that the participant would use as the basis for filling out a Form 1 . 
There were no overlapping items between Assessments A and B. Participants were randomized to 
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the order of assessments in a balanced design such that half of the participants received Assessment 
A as the pretest (with Assessment B as the posttest), and half received Assessment B as the pretest 
(with Assessment A as the posttest). All multiple choice questions in the assessment of knowledge 
and clinical reasoning were worth 1 point each. For multiple-select questions with more than one 
correct response, the following marking criteria were used: each question was worth 1 point; if the 
participant did not follow directions in terms of choosing too many answers (when a single correct 
response was indicated), then it was scored as a zero. If there were three possible answers, selecting 
all three correct answers was scored as 1 point; selecting two of three correct answers was scored as 
0.667 points; and selecting one of three correct answers was scored as 0.333 points. 

Two raters (SG and AJL), both psychiatrists with extensive experience reviewing Form 1’s created a 
scoring method for the Form 1, breaking down the various elements of the form into regions, each 
region being worth 1 point if filled out accurately. A correctly filled out Form 1 was scored out of a 
maximum of 12 points. Sample forms were marked blinded for reviewer calibration. The Intraclass 
Correlation (ICC) as the measure for inter-rater reliability was 0.98. In total, the maximum number of 
points for the assessment of knowledge was 5; the maximum score for the assessment of clinical 
reasoning was 5; and the maximum for the Form 1 was 12 points. All assessments were completed 
on paper, with subsequent data entry by a research assistant blinded to participant information and 
study hypothesis. The primary outcomes of interest were the change between pre- and posttest 
performance on tests of knowledge, clinical reasoning and ability to accurately fill out the Form 1 of 
the Mental Health Act based on a written scenario. 

3.6 Data analysis 

All data were anonymised and coded, and transferred into SPSS (version 17 for MacOS) for statistical 
analysis. Inter-rater reliability was determined using the Intraclass Correlation. Repeated measures 
analysis of variance (ANOVA) was conducted with two within-subjects measures each with 2 levels 
(pretest vs posttest and rater 1 vs rater 2), and two between-subjects factors of trainee type with 3 
levels (medical student vs non-psychiatry resident vs psychiatry resident) and the specific order of the 
assessment test performed with 2 levels: (assessment order A/B vs assessment order B/A). We 
elected to include trainee type as a factor in the ANOVA as we hypothesised that psychiatry residents 
might have more experience and training in how to fill out the Form 1 . 

3.7 Ethics 

Approval was obtained from both hospital and university research ethics boards, and all participants 
provided written informed consent. Identifying information was collected but kept separately from 
assessment data, and subject participation was voluntary. There were no course credits or 
evaluations related to participation. 

4. Results 

7 undergraduate medical students (clinical clerks) and 15 postgraduate resident physicians (n=22 
total) were recruited and enrolled in the study, and complete follow-up data were recorded for all 
participants. Of the 15 residents, 5 were recruited from psychiatry, 1 from emergency medicine, 1 
from surgery, and 8 from internal medicine. For all outcome measures of knowledge, clinical 
reasoning, and skill at filling out the Form 1 there was a statistically significant improvement between 
pretest and posttest performance (knowledge, F(1 ,21 ) 54.5, p<0.001 ; clinical reasoning, F(1 ,21 ) 9.39, 
p=0.006; Form 1 skill, F(1 ,21 ) 15.7, p=0.001). Further analysis showed no significant differences or 
interactions with other variables such as between raters, the order of assessment, or trainee type. 
See Table 1 and Figure 1 for details. 


Table 1 : Pretest vs posttest results for each assessment measure 


Measure 

Pretest mean (SE) 

Pretest 95% 
Confidence Interval 

Posttest mean (SE) 

Posttest 95% 
Confidence Interval 

Knowledge 
(out of 5) 

3.0 (.269) 

2.4-3. 6 

4.6 (.140) 

4. 3-4. 9 

Clinical 
reasoning 
(out of 5) 

3.3 (.234) 

2. 9-3. 8 

4.1 (.204) 

3. 7-4. 6 

Form 1 
(out of 12) 

9.4 (.41 6) 

8.5-10.3 

1 1 .2 (.336) 

10.4-11.9 
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Figure 1: Pretest vs posttest scores for each outcome of knowledge (out of 5), clinical reasoning (out 
of 5), and filling out the Form 1 (out of 12). Error bars represent standard error of the mean. 
For all outcome measures there was a statistically significant improvement between pretest 
and posttest performance. ** p<0.01 ; *** p< or =0.001 

5. Discussion 

In this study, we looked at the efficacy of an e-learning module to teach physicians how to complete a 
Form 1, by comparing baseline knowledge and skills with posttest performance. For all outcome 
measures of knowledge, clinical reasoning, and skill at filling out the Form 1 there was a statistically 
significant improvement between pretest and posttest performance following a 45-minute online 
educational intervention. 

5.1 Limitations 

Our study has several limitations. First, it was an efficacy study under laboratory conditions; it is 
unclear whether effectiveness would be the same under less controlled conditions, where there is 
often attrition in online training programs (Clark and Mayer 2007). Second, there was only an 
immediate posttest, with no delayed posttest to assess far transfer of skills beyond the hour of the 
study. Third, there was no attempt to evaluate whether or not the learning could be applied in the real- 
world clinical setting. Fourth, there was only one Form 1 graded for each assessment, with slightly 
different scenarios (a community-based scenario vs hospital-based scenario), whereas it might have 
been better to have had additional Form 1’s assessed. Fifth, pretest/posttest studies typically show 
large effect sizes, as we saw in our study. These limitations taken into consideration, our study 
proving efficacy under tightly-controlled conditions is still a valuable pre-requisite before conducting 
additional effectiveness and quality assurance experiments. 

5.2 e-Learning for skills training 

In a recent meta-analysis and systematic review, Cook et al analysed both e-learning vs no- 
intervention controls, as well as e-learning vs alternate instructional media. No intervention control 
studies looking at skills outcomes (n=16) showed a pooled effect size of 0.85, but there was large 
inconsistency across trials (Cook 2008). Interventions with practice exercises showed a significantly 
higher effect size, but other instructional design variables such as interactivity, repetition, or having an 
associated discussion component were not shown to affect outcomes. FHalf of the studies used a 
posttest only design, with only 3 of the 16 studies using a single-group pretest-posttest design as we 
used in our study. 11 of the studies had a similar participant pool of either medical students or 
physicians. In the meta-analysis of studies using e-learning compared with non-Internet interventions, 
only 12 studies reported skills outcomes. The pooled effect size for these studies was only 0.09, 
indicating only a small effect for the use of the Internet as the training medium vs alternate 
instructional media. 



www.ejel org 


243 


ISSN 1479-4403 


Electronic Journal of e-Learning Volume 7 Issue 3 2009, (239 - 246) 


Our intervention had practice exercises for the clinical reasoning cases, but not for the skills outcome 
of filling out the Form 1. However, there were several worked examples demonstrating how to fill out 
the form and the improvement in outcome is consistent with a body of literature that suggests that 
worked examples can be as effective as practice exercises for training learners with low prior 
knowledge (Clark and Mayer 2007; Clark 2008; Clark 2003). Moreover, worked examples are often 
easier to create for the program developer, and they can be more time efficient for the learner. We 
employed best practices in the instructional design of the e-learning intervention, consonant with the 
key evidence-based principles of multimedia learning (Mayer 2001 ; Clark and Mayer 2007). However 
we did not conduct a ‘within-media’ comparison of alternate instructional designs in this study, as in 
some of our previous work (Levinson 2007). In a systematic review and meta-analysis of instructional 
design variations within Internet-based learning, Cook et al found modest effect sizes for the use of 
the instructional design elements of high interactivity, practice exercises, feedback and repetition on 
improving learning outcomes (Cook 2009). However, only 3 of the 52 studies reviewed reported on a 
skills outcome. It is possible that we might have been able to achieve even greater efficacy with an 
alternate design (e.g. more practice exercises). In future experiments, we plan to explore the use of 
performance support or job aids to help physicians fill out the Form 1 in the workplace setting. Given 
the legal importance of the Form - but the infrequency for some physicians of completing the task - a 
job aid at point-of-care may prove to be more effective and pragmatic than educational modules 
(Rossett and Schafer 2007). Assessment of long-term retention of the skills will also be of interest. 
We hope to conduct additional evaluation and Quality Improvement protocols looking at a modified 
Kirkpatrick ‘Four Levels’ (Kirkpatrick and Kirkpatrick 2006) to better measure physician behaviour 
change and return-on-investment of the training. 

5.3 Conclusions 

Despite these issues and limitations we have developed a highly efficacious and relatively efficient 
intervention in a medium (e-learning) that can be easily disseminated to trainees and physicians 
across the province of Ontario (or modified for other jurisdictions) in an under-represented and 
important topic. Under laboratory conditions, this e-learning module demonstrated substantial efficacy 
for training medical students and residents on the theory and practice of accurately completing the 
Form 1 of the Mental Health Act. E-learning may prove to be an efficient and cost-effective medium 
for training of physicians on this important medico-legal aspect of care. Further research is required to 
look at longer-term impact of training and broader implementation strategies across the province for 
medical trainees and practicing physicians. 
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